Multi-View CNN-LSTM Architecture for Radar-Based Human Activity Recognition

In this paper, we propose a Multi-View Convolutional Neural Network and Long Short-Term Memory (CNN-LSTM) network which fuses multiple “views” of the time-range-Doppler radar data-cube for human activity recognition. It adopts structure convolutional neural networks to extract optimal frame based features from time-range, time-Doppler range-Doppler projections data-cube. The CNN models are trained using an unsupervised Auto-Encoder (CAE) topology. Afterwards, pre-trained parameters encoder fine-tuned intermediate representations, subsequently aggregated via LSTM sequence classification. temporal correlation among views is explicitly learned by sharing weights across different views. Moreover, range Doppler energy dispersion difference as input CNN-LSTM models. Furthermore, investigate use target tracking auxiliary side information. proposed model on datasets collected in both cluttered uncluttered environments. For validation, independent test dataset, with unseen participants, environment was collected. Fusion improves generalization 5%, yielding overall Macro F1-score 74.7%.